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ABSTRACT 

Several  widely  used  uniform  random  number  generators  have  been  ex- 
tensively subjected  to  three  commonly  used  statistical  tests  of  uniformity 
and  randomness.   The  object  was  i)  to  examine  the  power  of  these  statis- 
tical tests  to  discriminate  between  "good"  and  "bad"  random  number  gener- 
ators, ii)  to  correlate  these  results  with  recently  proposed  mathematical 
characterizations  of  random  number  generators  which  might  also  be  useful 
in  such  a  discrimination,  and  iii)  to  examine  the  effect  of  shuffling  on 
the  random  number  generators . 

Briefly  the  results  show  that  the  commonly  used  runs  test  has  virtu- 
ally no  power  to  discriminate  between  "good"  and  "bad"  generators,  while 
serial  tests  perform  better.   Also  shuffling  does  help,  although  much 
more  needs  to  be  done  in  this  area.   And  finally,  there  is  some  utility 
to  the  mathematical  characterizations,  but  many  unanswered  questions. 
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Abstract 

Several  widely  used  uniform  random  number  generators  have  been  extensively  subjected 
to  three  commonly  used  statistical  tests  of  uniformity  and  randomness.   The  object 
was  i)  to  examine  the  power  of  these  statistical  tests  to  discriminate  between 
"good"  and  "bad"  random  number  generators,  ii)  to  correlate  these  results  with 
recently  proposed  mathematical  characterizations  of  random  number  generators  which 
might  also  be  useful  in  such  a  discrimination,  and  iii)  to  examine  the  effect  of 
shuffling  on  the  random  number  generators. 

Briefly  the  results  show  that  the  commonly  used  runs  test  has  virtually  no  power 
to  discriminate  between  "good"  and  "bad"  generators,  while  serial  tests  perform 
better.   Also  shuffling  does  help,  although  much  more  needs  to  be  done  in  this 
area.   And  finally,  there  is  some  utility  to  the  mathematical  characterizations, 
but  many  unanswered  questions. 


1.   INTRODUCTION 

The  generation  of  pseudo-random  numbers  has  been 
the  subject  of  literally  hundreds  of  papers  in  the 
computing  and  simulation  literature  (see  the  bibli- 
ography by  Nance  and  Overstreet  [13]).   By  far  the 
most  popular  method  of  pseudo-random  number  gener- 
ation has  been  the  Lehmer  congruential  method: 


n+1 


A«X  +  C 
n 


(mod  P) 


(1) 


The  theory  underlying  the  implementation  and  use 
of  the  congruence  (1)  is  well  described,  see  e.g. 
Knuth  [5]. 

Recent  attention  has  been  given  to  alternatives  to 
the  Lehmer  congruential  method.   Most  notable  are 
the  feedback  shift  register  generators  [10,  15,  18 ] 
These  generators  are  based  on  irreducible  polyno- 
mials over  GF(2)  and  are  capable  of  producing 
extremely  long  sequences  relative  to  the  word  size 
of  the  computer. 

The  generation  of  pseudo-random  numbers  is  itself 
a  simulation,  that  is,  we  are  attempting  to  simu- 
late a  random  sequence  of  numbers.   As  in  any 
simulation  experiment,  we  apply  some  sort  of  test 
procedure  to  verify  how  well  our  simulation 
achieved  the  desired  goal.   Typically  runs  tests, 


serial  tests,  and  various  chi-square  tests  for 
independence  are  applied  to  relatively  short  sec- 
tions of  the  pseudo-random  sequence.   Many  of  the 
tests  have  been  outlined  by  Gorenstexn  [4]  and 
Knuth  [5] .   Recent  results  have  indicated  that  th 
power  of  these  classical  tests  to  detect  "poor" 
generators  is  suspect. 

We  now  have  mathematical  characterizations  of  pse 
random  sequences  which  should  help  one  to  discrim 
inate  between  generators.  Chief  amongst  these 
characterizations  are  the  spectral  structure  de- 
scribed by  Coveyou  and  MacPherson  [1]  and  the  lat 
tice  structure  test  advanced  by  Marsaglia  [11]  an 
Smith  [17]  for  congruential  generators.  There  ar 
also  presumably  better  statistical  tests  based  on 
the  properties  of  the  periodogram,  as  outlined  in 
Lewis,  Goodman,  and  Miller  [9].  All  of  these  tes 
are  more  sensitive  to  subtle  departures  from  rand* 
ness  than  some  of  the  older  tests,  both  mathemati 
and  statistical,  particularly  with  respect  to  the 
higher  dimensional  properties  of  the  pseudo-randoi 
sequences.  However,  their  interrelationship  is  n 
well  understood.  For  example,  Marsaglia  [11]  de- 
rides the  utility  of  the  Coveyou-MacPherson  work. 
Moreover,  in  a  real  sense  the  proof  of  the  puddin; 
(here  a  pseudo-random  number  generator)  is  in  the 
using  and  it  is  not  clear  how  well  the  mathematic. 
tests  predict  the  results  of  the  statistical  test; 
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i.e.  the  ability  of  the  sequences  to  simulate 
independent  uniform  random  numbers . 

In  this  paper  we  propose  to  examine  the  perform- 
ance of  six  pseudo-random  number  generators  on  the 
runs  test  and  serial  test  for  pairs  and  triples. 
Among  these  six  generators  are  two  known  to  be 
poor.   Testing  these  serves  to  demonstrate  the 
lack  of  discriminatory  power  of  some  of  the  statis- 
tical tests.   In  the  case  of  one  of  them  (RANDU) 
the  mathematical  characterizations  clearly  predict 
poor  performance.   The  idea  of  shuffling  the  pseu- 
do-random sequence  will  also  be  explored  as  a 
means  of  improving  the  statistical  performance  of 
a  generator.   This  is  a  relatively  new  and  untested 
method. 

2.   THE  GENERATORS 

The  six  generators  used  in  this  report  were  all 
designed  for  use  on  the  32-bit  word  IBM  System/360. 
Table  1  summarizes  the  pertinent  information  about 
each  of  the  six  generators. 

LLRANDOM  is  a  version  of  the  generator  reported  in 
Lewis,  Goodman  and  Miller  [9]  (see  Learmonth  and 
Lewis  [7]  for  details  of  LLRANDOM).  Other  versions 
of  the  Lewis,  Goodman  and  Miller  generator  are  used 
in  the  new  IBM  SL/MATH  package  and  in  the  IBM  ver- 
sion of  the  APL  language.   RANDU  is  the  uniform 
random  number  generator  provided  with  the  IBM 
Scientific  Subroutine  Package  [16]  which,  despite 
disclaimers  of  support  from  IBM,  is  still  unfor- 
tunately widely  used.   TAUS  is  a  feedback  shift 
register  generator  proposed  by  Tausworthe.   The 
present  implementation  was  patterned  after  Payne 
[15]. 

GFSR  is  a  FORTRAN  implementation  of  a  "generalized" 
feedback  shift  register  generator  by  Lewis  and 
Payne  [10].   The  algorithm  employed  in  GFSR  sim- 
plifies the  coding  of  the  feedback  shift  register 
and  generalizes  the  implementation  to  virtually 
any  word-size  machine. 


The  last  two  generators  listed  in  Table  1  were 
taken  from  the  package  "Super-Duper"  by  G.  Marsaglj 
The  novelty  of  "Super-Duper"  lies  in  the  fact  that 
it  consists  of  combining  (exclusive  OR'ing)  the 
results  of  both  a  Lehmer  congruential  generator  ant 
a  feedback  shift  register  generator.   For  one  case 
(Super-Duper)  we  use  this  combination  method.   For 
the  last  generator  we  used  only  the  congruential 
part  of  the  Super-Duper  package. 

2.1   SHUFFLING. 

The  sequences  produced  by  pseudo-random  number  gen- 
erators are  deterministic.  In  order  to  disguise 
this  pattern,  several  techniques  have  been  proposec 
to  reorder,  or  shuffle,  the  sequence  emanating  fror 
a  generator.  Marsaglia  and  Bray  [13]  have  proposec 
maintaining  a  table  of  pseudo-random  numbers  withii 
the  generator.  Two  independent  pseudo-random  num- 
bers are  generated  at  each  call.  The  first  number, 
appropriately  scaled,  is  used  as  a  random  index 
into  the  table.  The  tabled  value  is  then  returned 
with  the  number  from  the  second  sequence  replacing 
it  in  the  table. 

Another  procedure  mentioned  by  Tukey  [19]  is  to 
generate  blocks  of,  say  1024,  pseudo-random  numbers 
and  then  shuffle  them  according  to  some  random  per- 
mutation. After  all  or  some  of  these  numbers  have 
been  used,  generate  another  block  and  shuffle  them 
according  to  the  same  permutation. 

Marsaglia's  [11]  method  of  exclusive  OR'ing  the  oul 
put  of  a  Lehmer  congruential  generator  and  a  feed- 
back shift  register  generator  is  also  aimed  at 
breaking-up  the  basic  deterministic  pattern  of  the 
Lehmer  generator.   It  produces  a  perfect  (in  the 
sense  of  unit  cell  volume)  lattice  structure. 

For  the  purposes  of  the  testing  described  here,  the 
Marsaglia  and  Bray  method  was  used  with  five  of  the 
six  generators,  the  exception  being  GFSR.   A  table 
size  of  128  was  chosen  and  two  sequences  of  the 
same  generator  were  used,  each  with  a  different 
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NAME 

TYPE 

MULTIPLIER 

MODULUS 

LLRANDUM 

LEHMER  CONGRUENTIAL 

75  =  16807 

231-1 

RANDU 

LEHMER  CONGRUENTIAL 

216  +  3  =  65539 

232 

TAUS 

FEEDBACK  SHIFT 
REGISTER 

x31  +  x13  +  1 

GFSR 

GENERALIZED  FEEDBACK 

x124  +  x37  +  1 

SHIFT  REGISTER 

PERIOD 


LEHMER  CONGRUENTIAL 
WITH  FSR 


SUPER-DUPER 

(CONGRUENTIAL   LEHMER  CONGRUENTIAL 
ONLY) 


69069 

x17  +  x15  +  1 


69069 


.32 


.32 


!31 
.29 


.31 


.29 


LANGUAGE 


-  2     SYSTEM/ 360  ASSEMBLER 

FORTRAN  IV 

-  1     FORTRAN  IV 


2124  -  1    FORTRAN  IV 


246  -  229   SYSTEM/ 360  ASSEMBLER 


SYSTEM/ 360  ASSEMBLER 


TABLE  1.   SUMMARY  OF  THE  SIX  GENERATORS  TESTED 


starting  value.   To  be  consistent,  this  shuffling 
scheme  was  used  in  LLRANDOM  even  though  LLRANDOM 
has  the  capability  of  shuffling  directly  incorpor- 
ated into  the  generator.   The  shuffling  scheme  in 
LLRANDOM  requires  only  one  pseudo-random  number 
since  seven  bits  of  that  number  are  used  to  index 
the  table  of  128  (27=128) .   These  seven  bits  are 
quite  random  due  to  our  choice  of  modulus  (see 
Knuth  [5]  page  12).   Results  of  the  statistical 
testing  of  the  actual  shuffled  sequence  produced 
in  the  LLRANDOM  package  are  given  in  Appendix  A. 

As  will  be  seen,  the  results  of  shuffling  are  en- 
couraging when  measured  by  statistical  tests,  even 
for  poorly  constructed  generators.   However,  it 
is  not  entirely  clear  how  shuffling  changes 
the  lattice  structure  of  the  sequence,  and  this 
raises  important  questions. 


A  chi  square  statistic  is  computed  for  the  sample 
generated  as 

NH  -  E[NJ2 

(5) 


X27  = 


E[Nd] 


This  statistic  is  only  approximately  distributed  as 
a  chi  square  variate  with   7   degrees  of  freedom 
since  the  expected  values  in  each  cell  are  not 
equal,  the  last  two  cells  both  having  expected 
values  less  than  5.   Additionally,  there  is  a  cer- 
tain lack  of  independence  since  a  long  run  is  usu- 
ally followed  by  a  short  run. 

The  test  procedure  was  to  perform  this  runs  test  for 
samples  of  size  N  =  65,536.   One  hundred  replica- 
tions were  then  performed  using  different  starting 
values  (seeds) . 


3.   THE  RUNS  TEST 


3.2   RESULTS  OF  THE  TESTS 


Pseudo-random  number  generators  produce  sequences 
which  are  inherently  periodic  although  when  P   is 
prime  and  A  is  a  positive  primitive  root  of  P, 
as  in  LLRANDOM,  the  numbers  do  not  repeat  until  P 
have  been  generated  and  it  is  felt  that  this  helps 
reduce  local  cyclic  effects.   The  runs-up-and-down 
test  is  frequently  applied  to  pseudo-random  sequen- 
ces to  investigate  the  possibility  of  local  (less 
than  full  period)  cycles  or  stationary  dependence. 

Under  the  null  hypotheses  that  a  sequence  is  ran- 
dom the  expected  value  and  variance  of  the  number 
of  runs  of  given  lengths  in  a  sequence  of  length 
N  are  easily  derived.   Levene  and  Wolfowitz  [8] 
have  shown  that  for  the  observed  number  of  runs  of 
length  d,  N  ,   the  statistic 

Nd  -  E[Nd] 


{Var[N,]} 
d 


1/2 


(2) 


is  asymptotically  normally  distributed  with  mean 
0  and  variance  1  as  the  sample  size,  n,   tends 
to  infinity. 

The  literature  on  the  power  of  the  runs  test  is 
rather  sparse,  even  though  it  is  widely  recommended 
as  a  test.   The  only  significant  result  is  due  to 
F.  N.  David  [2]  which  shows  that  the  runs  test  has, 
asymptotically,  the  greatest  power  of  any  test  of 
randomness  against  the  alternative  of  first-order 
Markov  dependence  in  a  binary  sequence. 

3.1   TEST  FORMULATION. 

The  runs -up -and -down  test  used  here  is  patterned 
after  Kendall  and  Stuart  and  was  used  and  described 
fully  by  Lewis,  Goodman  and  Miller  [9].   Samples 
of  size  N  =  65,536  were  generated  and  runs  counted 
for  lengths   d  =  1,2,..., 7,   with  a  last  cell  d  =  8 
collecting  observed  runs  of  length  8  or  more. 
The  expected  number  of  runs  is  given  by 

2(n-d-2)(d2+3d+l) 


E[Nd]         (d+3)! 


E[N 


2n  -  7 


8+J 


"  I 


d  =  1,2,. ..,7  (3) 


(4) 


d=l 


Samples  of  one  hundred  chi  square  statistics  (5) 
were  obtained  from  each  of  the  six  generators.   An 
additional  sample  of  one  hundred  was  taken  for  each 
of  the  generators,  except  GFSR,  with  shuffling 
implemented.   Each  set  of  one  hundred  was  considered 
to  be  a  sample  from  the  distribution  of  this  runs 
test  statistic  (5) . 

A  preliminary  analysis  of  the  sample  data  showed 
that  for  some  starting  values,  all  six  of  the  gen- 
erators would  fail  the  runs  test  at  a  5%  level  based 
on  a  chi  square  distribution  with   7  degrees  of 
freedom.   Even  when  shuffled,  each  generator  pro- 
duced values  of  the  test  statistic  which  were  either 
too  high  or  too  low  for  acceptance.   The  question  is 
however,  are  the  numbers  of  rejections  consistent 
with  the  chi-square  distribution  theory,  and  if  not, 
can  we  simulate  the  true  (null)  distribution  with 
our  results.   The  answer  to  the  first  question  is 
that  no  generator  produced  results  which  were  gross] 
different  from  that  predicted  by  the  chi-square 
distribution  theory. 

Thus,  to  further  assess  the  ability  of  the  runs  test 
to  discriminate  "bad"  generators  from  "good"  gener- 
ators, it  was  decided  to  compare  sample  distribu- 
tions of  the  runs  test  statistic  (5) .   A  two-sample 
Kolmogorov-Smirnov  test  was  performed  on  all  pairs 
of  samples  (of  size  100).   The  two-sample  K-S   test 
is  distribution-free  and  we  avoided  the  problem  of 
having  to  specify  an  exact  null  distribution  for 
the  statistic  (5) .   A  two-sample  K-S   test  subrou- 
tine was  programmed  using  the  algorithm  presented 
by  Kim  and  Jennrich  [6]. 

Table  2  summarizes  the  results  of  these  tests  with 
the  generators  not  shuffled.   The  pair  of  values 
given  correspond  to  the  sample  K-S   criterion, 
c/(m«n),   and  the  Pr{D(m,n)  >  c/(m-n)},  respectively 
The  results  indicate  that  RANDU  produces  runs  test 
statistics  which  are  not  distributionally  commen- 
surate with  the  other  five  generators.   This  was  to 
be  expected  since  RANDU  is  one  of  the  two  "poor" 
generators  among  the  six,  but  the  indications  are 
not  strong.   The  somewhat  surprising  result  is  that 
TAUS  appears,  except  for  the  case  of  RANDU,  to  be 
similarly  distributed  with  the  remaining  four 


generators.   TAUS  is  the  other  suspect  generator 
referred  to  above.   This  is  because  the  results  of 
Toothill,  Robinson  and  Adams  [18]  indicate  that  a 
feedback  shift  register  generator  with  the  primi- 
tive trinomial  used  in  TAUS  could  not  comply  with 
the  moment  and  marginal  results  of  Levene  and  Wolf- 
owitz  [8]. 

Table  3  presents  results  of  a  similar  test  using 
the  shuffled  samples  (with  the  exception  of  GFSR 
which  was  not  shuffled) .   With  the  type  1  error  at 
5%,  all  generators  appear  to  be  distributionally 
commensurate.   This  result  is  encouraging  since  it 
was  hoped  that  shuffling  would  improve  the  perform- 
ance of  relatively  poor  generators  such  as  RANDU. 


variate  with  seven  degrees  of  freedom  are  y2 

,,  „         o  *7,0.9I 


=  14.07  and 


X7  0.99  =  18-48  respectively, 


Although  the  runs  test  has  been  used  frequently  ir 
the  testing  of  pseudo-random  number  generators,  w« 
feel  now  that  the  test  is  of  very  doubtful  use.  ~ 
The  lack  of  distributional  theory  concerning  the 
runs  test  statistic  (5) ,  combined  with  its  empiri- 
cal lack  of  ability  to  discriminate  TAUS  leads  us 
to  conclude  that  the  runs  test  should  not  be  em- 
ployed in  testing  pseudo-random  number  generators. 
We  hope  we  have  laid  it  to  rest  forever.   It  is 
interesting  to  note  that  as  late  as  1971  Smith  [1/ 
in  an  excellent  paper,  recommended  a  generator  on 
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MARSAGLIA 
(CONGRUENTIAL) 


LLRANDOM 
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MARSAGLIA 
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MARSAGLIA 
(CONGRUENTIAL) 


23   .0062 


,13   .2820 


,21   .0156 


,17   .0783 


19   .0364 
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7021 


,17  .0783 

.18  .0539 

.08  .8154 

,12  .3682 


GFSR 


.09   .7021 


24   .0038 


,13   .2820 


,16   .1112 


15   .1549 


TABLE  2.   K0LM0G0R0V-SMIRN0V  TWO  SAMPLE  TEST  ON  RUNS  TEST  STATISTICS, 
SAMPLE  SIZE  =  100.     NOT  SHUFFLED. 


LLRANDOM 
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TAUSWORTHE 
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MARSAGLIA 
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.14  .2112 

.18  .0539 

.14  .2112 

.13  .2820 


GFSR 


.13  .2820 

.17  .0783 

.16  .1112 

.11  .4695 

.11  .4695 


TABLE  3.   KOLMOGOROV-SMIRNOV  TWO  SAMPLE  TEST  ON  RUNS  TEST  STATISTICS. 
SAMPLE  SIZE  =  100.   ALL  GENERATORS  EXCEPT  GFSR  ARE  SHUFFLED. 


As  a  last  test,  the  samples  of  shuffled  data  from 
five  of  the  six  generators  (the  exception,  again, 
was  GFSR)  were  pooled  to  form  a  sample  of  500  from 
the  distribution  of  the  runs  test  statistic.   A 
K-S   test  was  then  performed  matching  the  individ- 
ual samples  of  100  (not  shuffled)  against  this  com- 
posite.  The  results  showed  only  RANDU  to  be  dis- 
tributionally different,  again  demonstrating  the 
poor  power  of  the  runs  test.   Another  result  is 
that  we  have  a  simulation  of  size  500  of  the  dis- 
tribution of  the  statistic  (5)  under  (hopefully) 
the  null  independence  hypothesis.   For  reference 
the  order  statistic  estimate  of  the  0.95  quantile 
is   15.82;  that  for  0.99   is   22.05.   For  refer- 
ence the  corresponding  quantiles  for  a  chi-square 


the  basis  of  its  lattice  structure  and  then  said 
"To  check  it,  we  carried  out  what  is  generally 
recognized  as  a  sensitive  test  of  uniformity  and 
independence  of  a  sequence,  namely  the  runs  test." 

4.   THE  SERIAL  TEST 

In  testing  pseudo-random  sequences,  it  is  useful  t 
test  the  uniformity  of  successive  but  not  necessar 
ly  contiguous  numbers  taken  as  k-tuples  for  k  = 
2,3,...   A  standard  test  employed  is  the  serial 
test. 

Taking  the  sequence  {U.}   of  uniform   (0.0,1.0) 
deviates  from  a  generator  one  divides  the  k 


dimensional  unit  hypercube  into  r   smaller  equi- 
sized  hypercubes.   For  binary  computers  r   is  usu- 
ally taken  to  be  a  power  of   2  and  determines  the 
number  of  (leading)  bits  from  each  number  which 
will  be  tested.   The  k-tuples  themselves  may  be 
taken  as  overlapping  or  nonoverlapping,  e.g.  the 
sequence  of   2-tuples   {U^  },{U  tU.},{U  U,}   or 

the  sequence  of   2-tuples   {U.U.} ,{U_U. } ,{UC ,u\} , 

1   Z       J  H       JO 

...   In  a  sample  of  size  n,   the  expected  number 
in  each  of  the  small  hypercubes  is  n/r^  under  the 
null  hypothesis  of  multidimensional  uniformity. 
The  following  chi-square  test  statistic  may  then 
be  computed : 


Sk  = 


k   r        r 

Jj-l     jk-l    V  J1»J2......3k     / 


(6) 


where 


is  the  observed  number  of  k- 


l1,J2,- 


tuples  in  each  small  hypercube.   For  the  case  of 
nonoverlapping  k-tuples,   S   is  distributed  asymp- 
totically as  chi  square  with  r^  -  1  degrees  of 
freedom.   For  overlapping  k-tuples,  a  correction, 
due  to  Good  [3],  involving  a  chi  square  test  for 
uniformity  must  be  included  for  S   to  be  approx- 
imately chi  square. 

The  desirability  for  k-dimensional  uniformity 
stems  from  the  fact  that  in  many  stochastic  variate 
generation  algorithms,   k-tuples  of  uniforms  are 
used  together  to  form  the  desired  variate.   For 
such  an  application,  nonoverlapping  k-tuples  would 
be  tested.   The  case  for  using  overlapping  inter- 
vals in  a  serial  test  is  somewhat  less  clear. 
Aside  from  the  distributional  problems  with  the 
test  statistic,  it  is  hard  to  imagine  a  simulation 
which  would  require  overlapping  k-tuples,  implying 
that  some  uniform  deviate  would  be  used  more  than 
once.   The  serial  test  when  applied  to  overlapping 
intervals,  however,  can  be  considered  as  a  test  for 
serial  independence  in  a  pseudo-random  sequence. 
Like  the  product  form  of  the  serial  correlation 
(autocorrelation)  statistic,  this  form  of  the  ser- 
ial test  should  be  sensitive  to  lagged  serial 
dependence  in  the  pseudo-random  sequence. 

4.1  TEST  FORMULATION 

For  samples  of  size  N  =  65,541,   one  hundred 
samples  of  the  test  statistic  (6)  were  computed  for 
each  of  the  six  generators.   The  tests  were  repeated 
for  five  generators  with  shuffling  implemented. 
Two  forms  of  the  test  were  employed:   the  first  set 
involved  examining  pairs  of  successive  overlapped 
numbers  and  the  second  form  involved  overlapped 
triples.   In  both  cases  the  first  four  bits  were 
used . 

The  successive  deviates  were  also  lagged  for  j  = 
1,2,..., 6,  so  that, for  example,  for  pairs  we  took 
contiguous  numbers,  numbers  one  apart,  ...  ,  num- 
bers five  apart  (lag  6) . 

4.2  TEST  RESULTS 

Since  the  use  of  overlapped  intervals  invalidates 


comparing  moments  from  the  sample  statistics  (6) 
with  expectation7  from  a  chi  square  distribution, 
we  went  directl-  to   the  two  sample  Kolmogorov-Smir- 
nov  test  for  distributions.   Table  4  summarizes  the 
results  for  the  serial  test  for  pairs  on  the  six 
generators  without  shuffling.   Only  the  first  lag 
is  shown  here  for  brevity. 

The  results  appear  inconclusive.   LLRANDOM  has  been 
rejected  in  spite  of  published  results  ([9]  and  [11 
indicating  good  two-space  properties  for  this  gen- 
erator.  In  [9]  LLRANDOM  was  tested  by  comparing 
the  test  results  to  the  x2   (40  degrees  of  freedom 
distribution,  since  no  true  distribution  was  known. 
It  may  be  that  what  is  showing  up  here  is  the  rela- 
tively poor  2-lattice  structure  of  the  LLRANDOM 
generator,  as  ^hown  in  Table  8a,  indicating  corre- 
lation between  the  predictions  of  the  statistical 
and  mathematical  tests. 

Shuffling  was  then  applied  with  little  discernible 
improvement.   Pairs  of  generators  are  still  rejecte 
as  shown  in  Table  5,  but  the  shuffled  LLRANDOM  is 
not  singled  out.   We  return  to  this  later. 

In  triples,  Table  6  summarizes  the  results  for  the 
generators  without  shuffling.   Only  the  first  lag 
is  shown.   For  the  triples  this  means  taking  three 
successive  and  contiguous  pseudo-random  numbers, 
i.e.   {U1U2U3},{U2U3U4),{U3U4U5},...   The  only  sig- 
nificant result  here  is  that  RANDU  fails  completely 
and  the  departures,  as  predicted  by  the  lattice 
structure  and  wave  number  tests,  are  gross.   Al- 
though the  distribution  of  the  test  statistic  (6) 
is  not  chi  square  due  to  the  overlapping  intervals, 
the  expected  value  of  (6)  is  still  r*  -  1  or 
4095.   while  the  test  statistics  for  the  other  five 
generators  were  within  two  standard  deviations  of 
this  expectation,  RANDU 's  sample  average  test  sta- 
tistic was  28,787!   We  expected  RANDU  to  perform 
badly  here  since  RANDU  is  known  to  be  poor  in  three 
space. 

With  shuffling  implemented,  the  results  in  Table  7 
indicate  that  all  of  the  generators,  including 
RANDU  are  distributionally  commensurate.   Thus  thei 
is  a  very  substantial  improvement  in  this  property 
of  the  sequences  induced  by  shuffling. 

5.   CONCLUSIONS  AND  RECOMMENDATIONS 

Many  of  today's  statistical  questions  are  beinj 
answered  through  large-scale  simulation.   The  gen- 
eration of  good  pseudo-random  deviates  for  simula- 
tion and  Monte  Carlo  experiments  is  of  prime  impor- 
tance.  Unfortunately  many  generators  are  being  usi 
whose  statistical  properties  make  them  a  hindrance 
rather  than  an  aid  in  such  experiments.   Through 
personal  communication,  we  have  heard  from  several 
researchers  who  have  had  their  experiments  stymied 
due  to  poor  pseudo-random  number  generators.   The 
typical  computer  center  will  offer  several  genera- 
tors of  either  unknown  progeny  or  ones  whose  test- 
ing is  rather  weak.   RANDU,  which  is  one  of  the 
most  widely  used  generators,  is  typical  of  this 
situation.   It  is  supplied  by  IBM  as  part  of  the 
Scientific  Subroutine  Package  and  is  therefore 


LLRANDOM 

RANDU 

TAUSWORTHE 

MARSAGLIA 
(MIXED) 

MARSAGLIA 
(CONGRUENTIAL) 

TABLE  4. 


RANDU 


34   .0001 


TAUSWORTHE 


.38 

.11 


.0000 
.4695 


MARSAGLIA 
(MIXED) 


.33 
.08 
.13 


.0001 
.8154 
.2820 


MARSAGLIA 
(CONGRUENTIAL) 


.30 
.15 
.16 


.0002 
,1549 
.1112 


.12   .3682 


GFSR 


.38 

.13 
.07 


KOLMOGOROV-SMIRNOV  TWO  SAMPLE  TEST  ON  SERIAL  TEST  STATISTICS  FOR  PAIRS, 
SAMPLE  SIZE  =  100.     NOT  SHUFFLED. 


.0000 
.2820 
.9084 


.17   .0783 


,21   .0156 


RANDU 


TAUSWORTHE 

MARSAGLIA 
(MIXED) 

MARSAGLIA 
(CONGRUENTIAL) 

.20   .0241 

.10 

.5830 

.08 

.8154 

.10   .5830 

.11 

.4695 

.19 

.0364 

— 

.14 

.2112 

.25 

.0023 

— 



.14 

.2112 

GFSR 


LLRANDOM 

RANDU 

TAUSWORTHE 

MARSAGLIA 
(MIXED) 

MARSAGLIA 
(CONGRUENTIAL) 

TABLE  5. 


17   .0783 


20 

.0241 

11 

.4695 

10 

.5830 

.13   .2820 


,21   .0156 
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RANDU 


TAUSWORTHE 


MARSAGLIA 
(MIXED) 


MARSAGLIA 
(CONGRUENTIAL) 


GFSR 


LLRANDOM 

RANDU 

TAUSWORTHE 

MARSAGLIA 
(MIXED) 

MARSAGLIA 
(CONGRUENTIAL) 

TABLE  6. 


1.0   .0000 


.19 
1.0 


.0364 
.0000 


.15 

1.0 
.15 


,1549 
,0000 
.1549 


.12 
1.0 
.14 


,3682 
.0000 
.2112 


,12   .3682 


.12 
1.0 
.13 


KOLMOGOROV-SMIRNOV  TWO  SAMPLE  TEST  ON  SERIAL  TEST  STATISTICS  FOR  TRIPLES, 
SAMPLE  SIZE  =  100.     NOT  SHUFFLED. 


,3682 
,0000 
.2820 


.09   .7021 


12   .3682 


RANDU 


TAUSWORTHE 

MARSAGLIA 
(MIXED) 

MARSAGLIA 
(CONGRUENTIAL) 

GFSR 

.16   .1112 

.18   .0539 

.18   .0539 

.16   .1112 

.07   .9084 

.15   .1549 

.13   .2820 

.12   .3682 



.11   .4695 

.12   .3682 

.12   .3682 

LLRANDOM 

RANDU 

TAUSWORTHE 

MARSAGLIA 
(MIXED) 

MARSAGLIA 
(CONGRUENTIAL) 

TABLE  7. 


,12   .3682 


,17   .0783 


12   .3682 


10 
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convenient  to  use.   Several  reports  are  around 
which  cite  results  of  tests  for  uniformity,  runs 
tests,  and  serial  tests  all  confirming  RANDU's 
adequacy.   Users  of  other  computers  are  in  even 
worse  shape  since  their  random  number  generators 
are  rarely  documented  and  no  one  seems  to  know  from 
whence  they  came! 

The  random  number  generators  which  have  appeared  in 
the  literature  rarely  specify  what  tests,  if  any, 
have  been  performed.   In  the  IMSL  Library,  for 
instance,  not  only  are  there  no  test  results  cited, 
but  the  user  is  given  the  option  to  use  his  own 
multiplier.   Those  algorithms  which  do  acknowledge 
testing  usually  cite  the  runs  tests  and  serial 
^zests  investigated  here  (cf.  Smith  [17]  cited  above) 
tt  was  our  aim  in  this  paper  to  show  whether  these 
tests  are  sensitive  to  the  type  of  departures  from 
randomness  which  would  be  crucial  to  serious  simu- 
lation efforts. 

The  most  distinct  conclusion  one  can  draw  from  the 
results  given  in  this  paper  is  that  the  runs  test 
las  no  utility  as  a  test  for  randomness  in  pseudo- 
random number  generators. 

?he  serial  test  is  somewhat  more  sensitive  than  the 
runs  test  and  interesting  conclusions  can  be  drawn 
:rom  the  results  obtained  here,  as  follows: 

Ising  pairs  (Tables  4  and  5)  a  subtle  departure  in 
LRANDOM  was  evidenced,  but  nothing  startling 
showed  up  about  RANDU .   Shuffling  did  not  clear  up 
:he  situation  completely.   This  will  probably  be 
ised  as  the  basis  for  further  investigations  to 
liscriminate  among  shuffled  generators.   Using 
:riples,  gross  departures  in  RANDU  appear  (of  the 
lundred  samples,  all  were  rejected  at  a  5%  level 
ising  the  x2   approximation  for  the  distribution 
f  the  statistic)  and  after  shuffling,  all  genera- 
:ors  (including  RANDU)  are  statistically  commen- 
urate.   The  interesting  point  here  is  that  it  is 
tot  clear  how  much  the  shuffling  we  have  used 
hanges  the  lattice  structure  of  RANDU,  so  that  we 
tay  have  a   generator  which  passes  the  statistical 
ests  but  has  "poor"  structural  characteristics . 
hus  there  is  evidence  that  Marsaglia's  work  has 
y  no  means  provided  the  final  answer  on  congru- 
mtial  and  other  generators.  Also  more  needs  to  be 
:nown  about  the  effect  of  various  shuffling  schemes 
n  the  mathematical  characterizations. 

.  word  about  the  inherent  drawbacks  of  the  serial 

est  is  in  order  here: 

(i)   A  problem  with  any  serial  test  is  the  need 
for  a  great  deal  of  storage.   In  the  tests 
reported  here  we  were  limited  to  looking  at 
only  the  first  four  bits.   (This  takes  4096 
memory  cells;  five  bits  takes  8  times  as 
many.)   If  one  takes  only  the  first  four  bits 
one  can  hardly  have  a  sensitive  test  for  the 
whole  pseudo-random  word.   Also,  the  lattice 
structure  of  the  truncated  sequence  is  dif- 
ferent from  that  of  the  whole  word,  though 
the  structures  probably  mimic  each  other  to 
some  degree. 

ii)   There  is  a  great  problem  of  deciding  what 

lags  to  take  in  serial  tests.   In  the  tests 


described  above  RANDU  failed  miserably  on 
contiguous   (1,1)   triples,  but  not  on  other 
triples.   Thus,  if  one  looked  at  six  lags,  as 
we  have  done,  it  might  be  that  departures 
would  show  up  at  other  lags. 

Fortunately,  theoretical  results  obtained  by  Pro- 
fessor Murray  Rosenblatt  (personal  communication) 
indicate  that  it  is  primarily  the  tests  for  contig- 
uous pairs  which  will  show  up  departures  in  congru- 
ential  generators.   His  results  also  provide  some 
theoretical  basis  for  shuffling.   Another  possibilii 
is  to  go  to  tests  for  randomness  based  on  the  empir- 
ical spectrum  (Lewis,  Goodman  and  Miller  [9]).   Thi! 
test  combines  the  serial  tests  of  all  lags. 

For  comparative  purposes,  Table  8a  below  shows  the 
relative  lengths  of  sides  of  unit  cells  for  the 
three  Lehmer  congruential  generators  in  this  paper. 
These  numbers  were  computed  by  Marsaglia  [11].   Noti 
that  the  one  dimension  of  the  2-lattice  for  LLRANDOI 
is  relatively  large.   Also,  one  dimension  of  the 
three-lattice  of  RANDU  is  very  large. 


Generator   2-lattice   3-lattice 


4-lattice 


LLRANDOM     1,7.6 
RANDU        1,1. 
SUPER-DUPER   1,1.06 
(CONGRUENTIAL  ONLY) 


1,1.6,3.9   1,1.09,1.69,2.07 
1,1.1,1819   1,7.3,1856,1872 
1,1.14,1.29  1,1.14,1.16,1.30 

TABLE  8a. 


Table  8b  below  lists  the  Coveyou-MacPherson  wave 
numbers  for  LLRANDOM  and  RANDU.   These  numbers  were 
computed  and  kindly  supplied  to  us  by  Dr.  L. 
Richard  Turner  of  the  NASA  Lewis  Research  Center. 
As  Smith  [17]  points  out,  there  is  a  direct  rela- 
tionship between  the  2-lattice  and  the  2  dimensional 
wave  number.   Small  wave  numbers  correspond  to  poor 
n-space  properties. 


Dimension 

LLRANDOM 

RANDU 

2 

16807 

23172 

3 

638.9 

10.86 

4 

147.25 

10.77 

5 

67.21 

10.77 

6 

29.92 

7 

16.55 
TABLE  8b. 

Since  congruential  generators  have  inherent  flaws 
(see  Marsaglia  [12]),  the  idea  of  shuffling  the 
sequence  as  it  leaves  the  generator  seems  to  be  a 
reprieve  for  these  popular  generators.   Shuffling 
showed  improvements  in  the  tests  performed  here, 
particularly  for  RANDU  in  the  serial  test  on  triple; 
The  effect  of  the  various  types  of  shuffling  which 
have  been  proposed  will  be  investigated  in  exten- 
sions of  future  work. 

The  case  for  feedback  shift  register  generators  is 
encouraging,  but  still  unclear.   After  an  initial 
flurry  of  activity  in  this  field,  the  careful  work 
of  Toothill,  Robinson,  and  Adams  [18]  showed  that 
these  types  of  pseudo-random  number  generators  also 
had  their  flaws.  Without  a  careful  choice  of  prim- 
itive trimonial,  a  feedback  shift  register  generatoi 


could  be  shown  analytically  to  have  bad  statistical 
properties  (generally  runs  properties,  and  these 
may  be  irrelevant).   The  generator,  GFSR,  has  been 
constructed  more  carefully  than  TAUS  but  is  rather 
costly  in  both  initialization  time  and  computer 
storage,  making  its  utility  doubtful.   Until  more 
extensive  investigation  on  feedback  shift  regis- 
ters is  forthcoming,  we  cannot  conclude  that  they 
are  preferable  to  the  shuffled  congruential  gener- 
ators 
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The  following  table  presents  the  results  of  per- 
forming the  runs  test,  serial  test  for  pairs,  and 
serial  test  for  triples  using  LLRANDOM 's  own  shuf- 
fling scheme. 

As  before,  100  sample  values  of  each  test  statistic 
were  obtained  from  LLRANDOM  using  its  own  shuffling 
scheme.   This  sample  was  compared  with  the  samples 
of  the  six  generators  included  in  the  report  (in- 
cluding LLRANDOM  shuffled  according  to  the  Marsagli 
Bray  scheme) . 

The  results  of  the  two  sample  Kolmogorov-Smirnov 
tests  are  shown.   The  pair  of  values  in  each  cell 
are  the  observed  value  of  the  Kolmogorov-Smirnov 
criterion,   c/(m*n),   and  the  value  of  Pr{Dm  n  > 
c/(m«n)},   respectively.   With  the  exception  of  the 
LLRANDOM-Marsaglia  (congruential)  comparison  on  the 
serial  test  for  pairs,  the  results  indicate  LLRANDO 
to  be  distributionally  commensurate  with  the  other 
shuffled  generators  at  the  5%  confidence  level. 
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GENERATOR 


RUNS 


SERIAL  (PAIRS)      SERIAL  (TRIPLES) 


LLRANDOM 
(MARSAGLIA- 

-BRAY 

SCHEME) 

.10 

.5830 

.16 

.1112 

.10 

.5830 

RANDU 

.11 

.4695 

.08 

.8154 

.07 

.9084 

TAUS 

.08 

.8154 

.11 

.4695 

.09 

.7021 

MARSAGLIA 
(MIXED) 

.09 

.7021 

.10 

.5830 

.10 

.5830 

MARSAGLIA 
(CONGRUENTIAL) 

.13 

.2820 

.19 

.0364* 

.13 

.2820 

GFSR 


15   .1549 


.08   .8154 


,12.  .3682 


TABLE  A.   KOLMOGOROV-SMIRNOV  TWO  SAMPLE  TEST  COMPARING  LLRANDOM  (SELF-SHUFFLING) 

WITH  SIX  OTHER  SHUFFLED  GENERATORS. 
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